Not dead, just resting: The practical value of per publication citation indicators

Citation counts do not measure research impact under any reasonable definition of impact. Instead they are indicators of research impact or quality in the sense that in some disciplines, when appropriately processed, they tend to correlate positively and statistically significantly with human judgements of impact or quality. Although we have theory to suggest that, at least in the sciences, citations could tend to reflect the contributions of papers to the onward progress of scholarship (Merton, 1973), there are many examples to demonstrate that this is not always the case (MacRoberts & MacRoberts, 1989, 1996, 2010; Seglen, 1997). Moreover, all citation-based indicators are often wrong and misleading and for some fields are completely useless (e.g., music and art in Table A3 of: HEFCE, 2015a). Nevertheless, as argued below, as long as they have a significant positive correlation with human judgements then they can serve a useful role supporting peer review (see also: van Raan, 1998). They may even be a primary source of evidence in situations where the expense of peer review in comparison to its benefits makes it impractical or for cases where peer judgements are thought to be too biased. Thus, citation-based indicators should not be used as a primary arbitrator of research impact unless there are practical reasons why better alternatives are inappropriate. Peer review is, in general, a better solution and is the one used in the UK to direct a large amount of UK government research funding (£1.6 billion per year in 2014-15: Wilsdon, Allen, Belfiore, Campbell, et al., 2015) with the remainder being primarily distributed through competitive project grant applications. The Abramo and D’Angelo (2016) argument for a particular type of indicator should be analysed within this context: not as an attempt to construct a perfect measure of research impact but as an attempt to construct the best possible indicator of research impact that is based on citations. When counting citations to collections of papers in order to compare two or more heterogeneous sets of papers, it is important to normalise the counts in order to make the comparisons fairer. For example, medical research is cited more frequently than information science research and older papers have had longer to attract citations. One way of reducing biases is to divide each article by the average number of citations for other articles of the same type and publication year so that scores above 1 suggest that a publication has had above average impact, irrespective of its field and publication year. The Mean Normalised Citation Score (MNCS) (Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011a,b) extends this to sets of documents by calculating the arithmetic mean of the normalised citation score for each publication so that an MNCS score above 1 indicates that the collection of papers has had an above average citation impact, irrespective of the range of publication years, fields and document types. There have been suggestions for fine tuning aspects of this calculation, such as by replacing the arithmetic mean with the geometric mean for its better handling of highly skewed data (Fairclough & Thelwall, 2015), or to report instead the proportion of articles in the most highly cited percentile (e.g., top 10% or top 1%) (Tijssen, Visser, & Van Leeuwen, 2002) but neither of these affect the main discussion here. This is because the main issue is whether indicators like these that are calculated on a per publication basis can be useful rather than which of them is the best. The MNCS and related indicators are used in applied scientometrics as evidence to aid peer review and to support policy evaluations and decisions. In this context, Abramo and D’Angelo (2016) argue that they are “not worthy of further use or attention”. The root problem is that averaging citation counts across exhaustive sets of articles to be compared can lead to misleading results, as they clearly demonstrate. For example, if there are ten sets of articles, each of which contains all of the publications of a given Dutch physics research group 2011-2015 and the purpose is to decide how much government funding should be given to each group during 2016-2020 then groups would be